Feature spec: compute extensibility user experience #96
Conversation
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
… 2.5 Pro assisted in VSCode) Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
|
|
||
| ### Non-goals (out of scope) | ||
| <!-- What are we explicitly not trying to accomplish? --> | ||
| - Running the Radius control plane on a non-Kubernetes platform. |
There was a problem hiding this comment.
This is out of scope for now or forever?
There was a problem hiding this comment.
out of scope for now, added a link to the roadmap item here.
| } | ||
| } | ||
| // This container requests confidential compute | ||
| extensions: [ |
There was a problem hiding this comment.
Do we support extensions on UDTs? Wouldn't these be additional optional properties consumed as recipe inputs instead?
| 1. **Configure Environment-Specific Recipe Parameters (Optional)**: | ||
| * If recipes have parameters that need to be set globally for an environment, configure them using `rad recipe update` or during registration. | ||
|
|
||
| #### Packaging and Registering a "Recipe Pack" for an Environment: |
There was a problem hiding this comment.
This is a fantastic implementation, I like that version is already included.
Signed-off-by: Will Tsai <[email protected]>
|
|
||
| ### Non-goals (out of scope) | ||
| <!-- What are we explicitly not trying to accomplish? --> | ||
| - Running the Radius control plane on a non-Kubernetes platform. |
There was a problem hiding this comment.
Does this mean that with extensibility and core types as UDTs we shouldn't expect to be able to host UCP in a different platform?
|
|
||
| ### Positive user outcome | ||
| <!-- What is the positive outcome for the user if we deliver this, i.e. what is the value proposition for the user? Remember, this is user-centric. --> | ||
| As a platform engineer, I can confidently adopt Radius across my organization, knowing I can extend its capabilities to support any compute platform, secret store, or gateway my teams require, without waiting for built-in support or modifying Radius core. I can register, customize, and share recipes that define how Radius provisions these resources, ensuring consistency with our application definitions while maintaining flexibility in our infrastructure choices and adhering to organizational standards. This empowers my development teams to leverage Radius benefits regardless of the underlying infrastructure. |
There was a problem hiding this comment.
We are talking about containers, secret stores, and gateways a lot. Are these the main resource types that we are focusing on? Why are we not talking about other core resource types?
There was a problem hiding this comment.
Good point, we should include volumes, Will T said extenders are out of scope which I agree as I believe they'd just be replaced by new types/recipes.
There was a problem hiding this comment.
yup, added the inclustion of volume resources too now.
| } | ||
| } | ||
| - providers: { | ||
| - // provider configurations no longer hard-coded here |
There was a problem hiding this comment.
Are we saying we will add info such as subscription and respurce grpup to the bicep recipe itself?
There was a problem hiding this comment.
We already have a requirement from some users that we will need to move those to the environment
There was a problem hiding this comment.
On second thought, this shouldn't be encoded in the recipe. We can keep providers as is for now.
Signed-off-by: Will Tsai <[email protected]>
| * Once the Recipe Pack is registered, the environment is configured with all the specified recipes for the core UDTs. | ||
| * When applications are deployed to this environment, Radius automatically uses the corresponding recipes from the pack to provision `Applications.Core/containers@2025-05-01-preview`, `Applications.Core/gateways@2025-05-01-preview`, and `Applications.Core/secretStores@2025-05-01-preview` resources. | ||
| 1. **Manage and Update Recipe Packs**: | ||
| * Platform engineers can update the Recipe Pack manifest (e.g., point to new recipe versions, change default parameters) and re-register it. The CLI could offer options to overwrite existing registrations or manage versions of the pack within the environment. |
There was a problem hiding this comment.
We need to design/review/implement recipe versioning. Will that be a prerequisite before we work on recipe packs?
There was a problem hiding this comment.
I think Recipes versioning will be a separate feature that we don't need to implement as a part of Recipe Packs. However, we should include versioning in scope for the implementation of Recipe Packs from the get go, even if the underlying Recipes packaged within the Pack don't support versioning yet.
Clarified this in the goals/non-goals
| - } | ||
| ] | ||
| // This container requests confidential compute | ||
| + runtimes: { |
There was a problem hiding this comment.
Should the confidential compute property be part of the ACI-specific recipe? Or is this an interim case while we look at Radius supporting conf containers?
Signed-off-by: Will Tsai <[email protected]>
| 2. **Discover/Create Recipes**: | ||
| * Find community-provided recipes for desired platforms (e.g., ACI, AWS Fargate) from an OCI registry or Radius documentation. | ||
| * Or, create custom Bicep/Terraform recipes for `Applications.Core/containers@2025-05-01-preview`, `Applications.Core/gateways@2025-05-01-preview`, and `Applications.Core/secretStores@2025-05-01-preview` to target a specific platform or customize existing behavior. | ||
| 3. **Register Recipes for Core Types**: |
There was a problem hiding this comment.
Is there a way to constraint certain app files to be used against only certain recipe? Example: App with confidential container property set to true, should be allowed only on confidential recipe.
There was a problem hiding this comment.
Not currently, we discussed this as well as the ability to only allow certain apps to use certain options/properties in a recipe.
|
|
||
| ## Topic Summary | ||
| <!-- A paragraph or two to summarize the topic area. Just define it in summary form so we all know what it is. --> | ||
| Radius will be enhanced to support multiple compute platforms, secret stores, and gateway resources through a recipe-based extensibility model. This approach decouples Radius's core logic from platform-specific provisioning code. Core resource types (`containers`, `gateways`, and `secretStores`) will be implemented as User-Defined Types (UDTs) and will allow platform engineers to register Bicep or Terraform recipes for them. Radius will provide default recipes for Kubernetes and Azure Container Instances (ACI), but platform engineers can use, modify, or replace these to customize how Radius provisions resources to different environments, or to add support for entirely new platforms without requiring changes to Radius core. |
There was a problem hiding this comment.
Shouldn't core resource types include volumes?
There was a problem hiding this comment.
yes, it should. i'll add it.
| } | ||
| } | ||
| } | ||
| - providers: { |
There was a problem hiding this comment.
The provider stores the AWS and Azure details that get past to the recipe context right? https://docs.radapp.io/reference/context-schema/#environment So we'll need the AWS account and region, and Azure subscription and resource group in the environment somewhere.
There was a problem hiding this comment.
Yes, good point, this shouldn't be encoded in the recipe. We can keep providers in the environment as is for now.
| * The platform engineer configures `std-env` with recipes for `Applications.Core/containers@2025-05-01-preview` that deploy to standard compute (e.g., regular ACI or Kubernetes pods). These recipes might ignore or log a warning for confidential container requests if they don't support them. | ||
| * Example recipe registration (conceptual): | ||
| ```bash | ||
| rad recipe register std-container-recipe --environment std-env \ |
There was a problem hiding this comment.
I strongly believe we should not have any recipe names. This leaks the implementation abstraction into the developers' world. If the developer needs a different recipe, that should be a different resource type.
| // This container requests confidential compute | ||
| extensions: [ | ||
| { | ||
| kind: 'confidentialCompute' // Example extension kind |
There was a problem hiding this comment.
If there are additional properties needed to be set by the developer, that should be a different resource type. I do not think extension is the right thing here, nor do I understand how one would create an extension.
| 1. **Configure Environment-Specific Recipe Parameters (Optional)**: | ||
| * If recipes have parameters that need to be set globally for an environment, configure them using `rad recipe update` or during registration. | ||
|
|
||
| #### Packaging and Registering a "Recipe Pack" for an Environment: |
There was a problem hiding this comment.
A "resource pack" would be more useful. It would be a manifest of resource types and associated recipes. I would specify the resource pack during rad init.
There was a problem hiding this comment.
Could we define a set of resources and a set of recipes within a single environment resource, and deploy that? That could replace the need for recipe or resource packs.
There was a problem hiding this comment.
@brooke-hamilton see answer in a previous comment: https://github.com/radius-project/design-notes/pull/96/files#r2141274374
@zachcasper - I'm thinking Recipes and Resources would be bundled into the same pack - it seems pointless to have them separately as one wouldn't be used without the other. Thoughts?
| ``` | ||
| 1. **Package the Recipe Pack (Optional but Recommended)**: | ||
| * The manifest file and any local recipe files (if not using OCI URIs exclusively) could be bundled into an OCI artifact or a simple archive (e.g., .zip, .tar.gz) for easier distribution and versioning. | ||
| 1. **Register the Recipe Pack to an Environment**: |
There was a problem hiding this comment.
Register isn't the right verb here. We're configuring the environment. rad environment create --recipe-pack <location> seems more accurate.
There was a problem hiding this comment.
Good point, I've changed it to leverage the rad environment update my-env --recipe-pack ... command instead.
Signed-off-by: Will Tsai <[email protected]>
…com/willtsai/design-notes-radius into compute-extensibility-feature-specs
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
|
|
||
| * The default Recipe packs will have the `allowPlatformOptions` parameter set to `true` for core types like `Applications.Core/containers@2025-05-01-preview`, which allows platform engineers to punch through the Radius abstraction and use platform-specific options (e.g., `containerGroupProfile` for ACI) in their application definitions. | ||
|
|
||
| * By default, `rad init` will register recipes for Kubernetes provisioning for core types like `Applications.Core/containers@2025-05-01-preview` so that Radius may continue providing a local kubernetes experience out of the box. |
There was a problem hiding this comment.
so if a user wants to use a different compute platform, it's expected that the user registers an additional recipe pack for ACI, ECS, etc?
There was a problem hiding this comment.
nvm, just saw the point below. why don't we offer the choice between compute platforms as a user input for rad init?
There was a problem hiding this comment.
correct - initially i had considered expanding rad init experiences to be per platform, e.g. rad init kubernetes and rad init aci but ultimately decided against it as rad init is meant to be for a quickstart rather than setting up production like environments.
There was a problem hiding this comment.
potentially introduce ACI, ECS, etc. as a part of the rad init --full experience
There was a problem hiding this comment.
The Environment resource needs to have default recipes/recipe packs, not rad init. We want rad environment create to create a working environment. If there are no recipes, no one can deploy anything (unlike today).
There was a problem hiding this comment.
Great progress. I made several smaller comments, but the big changes I think we need to make are:
- A recipe pack should be a resource. Radius should store its configuration within Radius. Storing configuration outside the system introduces external dependencies and raises the risk of things being out of sync.
- secretStores should be renamed secrets.
- I still don't understand what is going on with volumes.
| description: "Recipe Pack for deploying to ACI in production." | ||
| recipes: | ||
| - resourceType: "Applications.Core/containers@2025-05-01-preview" | ||
| name: "aci-prod-container" # Optional: a friendly name for this recipe registration |
There was a problem hiding this comment.
We should not include a name for the recipe. Resource types intensionally do not have the ability for developers to specify the recipe name.
There was a problem hiding this comment.
Makes sense, because it'd be encouraged to use the default recipe. I'll remove the recipe name fields.
| > Note: if there are existing recipes registered for the same resource type in the environment, this command will overwrite them with the recipes defined in the pack. | ||
|
|
||
| 1. **List the Registered Recipes**: | ||
| * Use `rad recipe list --environment <env-name>` to see all registered recipes for core resource types in the specified environment. |
There was a problem hiding this comment.
Can we get an example here? I'm imagining:
$ rad recipe list
RECIPE PACK RESOURCE TYPE KIND LOCATION
kubernetesCore Applications.Core/containers Terraform https://github.com/project/radius/resource-type-contrib.git/recipes/core/kubernetes/containers?ref=v1.2.0
# Note the name argument is no longer required
$ rad recipe show --resource-type Applications.Core/containers
RECIPE PACK RESOURCE TYPE KIND LOCATION
kubernetesCore Applications.Core/containers Terraform https://github.com/project/radius/resource-type-contrib.git/recipes/core/kubernetes/containers?ref=v1.2.0
PARAMETER TYPE DEFAULT
allowPlatformOptions BOOL TRUE
There was a problem hiding this comment.
Good suggestion, added. However, a couple of notes:
- The allowPlatformOptions parameter is only applicable to the container resource type for now, so we'll only display it for that (will be blank for other resource type recipes)
$ rad recipe list RECIPE RECIPE PACK ALLOW PLATFORM OPTIONS TYPE RECIPE KIND RECIPE VERSION RECIPE LOCATION default Applications.Datastores/redisCaches bicep ghcr.io/my-org/recipes/azure/redis-azure:0.32 default azure-aci-pack true Applications.Core/containers bicep ghcr.io/my-org/recipes/core/aci-container:1.2.0 default azure-aci-pack Applications.Core/gateways bicep ghcr.io/my-org/recipes/core/aci-gateway:1.1.0 default azure-aci-pack Applications.Core/secretStores bicep ghcr.io/my-org/recipes/azure/keyvault-secretstore:1.0.0 - I'm keeping the behavior change for
rad recipe showout of scope for now, as we need to independently decide what is the required field ifnameis no longer required (e.g. does--resource-typebecome a required argument?)
|
|
||
| > This approach will require deprecation of the current [Azure KeyVault Radius Volumes resource type](https://docs.radapp.io/reference/resource-schema/core-schema/volumes/azure-keyvault/) where the Azure KeyVault will just be a Secret Store resource type provisioned using a recipe for Azure KeyVault that can be mounted to a Radius Container resource as a volume. | ||
|
|
||
| #### User Story 9: As an application developer, I want to create custom gateway resources and configure them to route traffic to my application containers, so that I can manage ingress traffic using my organization's preferred gateway solution: |
There was a problem hiding this comment.
What is a custom gateway? I think what you are saying is:
As a developer, I need to route ingress traffic to one of my containers on a particular port. My platform engineer has configured the platform to handle ingress traffic, so I only need to specify the route.
There was a problem hiding this comment.
I mean a gateway resource backed by an ingress controller of their choice (this is the "custom" part), e.g. a gateway resource backed by nginx ingress controller. I will clarify this.
There was a problem hiding this comment.
As an application developer, I want to add a gateway resource to my application that provivisions an ingress controller that I can use to route traffic to my application containers, so that I can manage ingress traffic using my organization's preferred ingress solution
is this better?
| Public endpoint http://1.1.1.1.nip.io/ | ||
| ``` | ||
|
|
||
| > Note: Contour is currently installed as a [hard dependency for Radius](https://github.com/radius-project/radius/blob/main/pkg/kubernetes/object.go#L27) to provide http routing for Kubernetes deployments. This change allows users to disable the Contour installation in their Radius environments if they want to use a different gateway solution, such as NGINX Ingress Controller or Traefik, by registering a custom recipe for `Applications.Core/gateways@2025-05-01-preview` that provisions the desired gateway solution. |
There was a problem hiding this comment.
| > Note: Contour is currently installed as a [hard dependency for Radius](https://github.com/radius-project/radius/blob/main/pkg/kubernetes/object.go#L27) to provide http routing for Kubernetes deployments. This change allows users to disable the Contour installation in their Radius environments if they want to use a different gateway solution, such as NGINX Ingress Controller or Traefik, by registering a custom recipe for `Applications.Core/gateways@2025-05-01-preview` that provisions the desired gateway solution. | |
| The developer UX is the same as today. However, several changes are implemented in the default gateway recipe implementation: | |
| 1. Contour is no longer installed and the `--skip-contout-install` option on `rad install` is removed | |
| 2. The gateway recipe, part of the kubernetesCore, implements L4 ingress only using a Kubernetes Service of type `LoadBalancer` | |
| 3. The gateway recipe, part of aciCore`, implements L7 ingress using Azure Application Gateway | |
| In the future, the gateway recipe in ecsCore will use AWS Application Load Balancer. | |
| Platform engineers can modify the gateway recipe to use other ingress options such as NGINX or Cilium on Kubernetes, or Azure Front Door with ACI. |
| Given: my platform engineer has set up a Radius environment with recipes registered for `Applications.Core/Containers@2025-05-01-preview` and `Applications.Core/volumes@2025-05-01-preview` resources. The volume recipe is configured to provision custom volumes based on the specifications determined by my organization. | ||
|
|
||
| 1. **Define a Custom Volume Resource**: | ||
| * The developer defines a custom volume resource in their application Bicep file that will be provisioned by the default recipe. |
There was a problem hiding this comment.
This user story doesn't quite make sense to me. How is this different from the volumes property on the container resource? Maybe if there were properties on the volume resource it would make more sense.
There was a problem hiding this comment.
Currently, there is no way to customize the type of storage resources that can be provisioned to be mounted as storage volumes on a container, e.g. mounting Azure Disk or AWS EBS as a volume on a container. The volumes property on the container today just mounts a volume using node-local storage and there's no option to swap that out with a storage service.
I've updated this user story to make it more clear, let me know if this helps.
Co-authored-by: Zach Casper <[email protected]> Signed-off-by: Will <[email protected]>
Co-authored-by: Zach Casper <[email protected]> Signed-off-by: Will <[email protected]>
Co-authored-by: Zach Casper <[email protected]> Signed-off-by: Will <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
…com/willtsai/design-notes-radius into compute-extensibility-feature-specs
Co-authored-by: Zach Casper <[email protected]> Signed-off-by: Will <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
| // providers property remains scoped to the environment | ||
| providers: { | ||
| azure: { | ||
| scope: '/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP_NAME>' |
There was a problem hiding this comment.
| scope: '/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP_NAME>' | |
| subscriptionId: <SUBSCRIPTION_ID> | |
| resourceGroup: <RESOURCE_GROUP_NAME> |
There was a problem hiding this comment.
We need to break this up into subscription ID (required) and resource group (optional).
See this issue. One user plans to map Radius Environments to subscriptions. Each resource that gets deployed via their Terraform modules gets put into a dynamically created resource group. So their recipes manage resource groups, not Radius.
| } | ||
| } | ||
| ``` | ||
| > Note: the `compute` property is removed in favor of Recipe configurations, with the Kubernetes namespace moved to the providers section. If no namespace is provided, then the namespace defaults to match the name of the environment as is [implemented today](https://github.com/radius-project/radius/blob/main/pkg/cli/cmd/env/create/create.go#L119-L121) |
There was a problem hiding this comment.
Do not do this. Platform engineers must be in control of Kubernetes namespaces, not Radius. If no namespace is specified, follow the Kubernetes convention of using the default namespace. Here is why:
Kubernetes namespaces are used in multi-tenant environments. Often, a team will have a namespace provisioned for them. The platform engineer then creates role assignments for a developer that grant permissions to the newly created namespace. If Radius is creating its own namespaces, the developer will not have access to it.
There was a problem hiding this comment.
makes sense, updated:
Note: the
computeproperty is removed in favor of Recipe configurations, with the Kubernetes namespace moved to the providers section. If a Kubernetes provider is specified but no namespace is provided, then the namespace defaults to 'default' to align with Kubernetes conventions. Note that this is changed from how it's implemented today.
| ``` | ||
| > Note: the `compute` property is removed in favor of Recipe configurations, with the Kubernetes namespace moved to the providers section. If no namespace is provided, then the namespace defaults to match the name of the environment as is [implemented today](https://github.com/radius-project/radius/blob/main/pkg/cli/cmd/env/create/create.go#L119-L121) | ||
|
|
||
| > If the `providers` property is not specified, default to `providers.kubernetes.namespace='<environment name>'` and show a warning to the user: "Since no provider is specified for the environment, the default provider is set to 'kubernetes' with namespace '<environment name>'." |
There was a problem hiding this comment.
| > If the `providers` property is not specified, default to `providers.kubernetes.namespace='<environment name>'` and show a warning to the user: "Since no provider is specified for the environment, the default provider is set to 'kubernetes' with namespace '<environment name>'." |
This error message is not necessary. If no provider is specified, the Environment should default to the default namespace.
There was a problem hiding this comment.
I think it's still valuable to show a warning to the user, but it'll be updated to reflect the defaulting behavior above:
If the
providersproperty is not specified, default toproviders.kubernetes.namespace='default'and show a warning to the user: "Since no provider is specified for the environment, the default provider is set to 'kubernetes' with namespace 'default'."
| - } | ||
| ] | ||
| // This container requests confidential compute | ||
| + platformOptions: { |
There was a problem hiding this comment.
Capturing the essence from my chat with Will and Shruthi:
I understood that when allowPlatformOptions is set and a developer specifies a property from the pod spec in the platformOptions, Radius is supposed to merge the properties before being sent to the recipe. If that's the case, what properties will the recipe author get?
In the example below, a Terraform configuration would get these variables:
var.context.resource.properties.application = "sensitiveApp"
var.context.resource.properties.container.image = "mycorp/sensitive-processor:v1.2"
var.context.resource.properties.container.ports.api.containerPort = "5000"
var.context.resource.properties.connections.cache.resource.properties.application = "sensitiveApp"
var.context.resource.properties.platformOptions = <object>
I'm not a fan of special pre-processing of recipes. But I also do not see how a merge of a map and the container schema can be handled in a declarative recipe. The objective of supporting punch throughs and using only declarative recipes seems in conflict.
| + // The Kubernetes metadata.labels at the environment level can be specified here, | ||
| + // this will apply to all containers deployed to this environment. | ||
| + // Developers can override in container definitions via platformOptions. | ||
| + metadata: { |
There was a problem hiding this comment.
With Terraform, this will likely result in a undeclared variable being specified. Radius takes all the parameters specified on a recipe and passes that to Terraform. Either my Terraform configuration has a metadata variable, or I will get the undeclared variable error.
The pattern of taking unstructured maps of data and trying to enrich the recipe is in conflict with the interface between Radius and recipes. Radius only passes values from the resource type's schema to the recipe. This is proposing to enrich that schema with unstructured data from the the resource definition AND the environment definition.
We want Radius to be declarative. We should avoid features which require pre-processing of untyped data.
|
|
||
| > This scenario highlights that the application definition remains consistent. The underlying infrastructure and specific compute capabilities (ACI standard vs. ACI confidential vs. Kubernetes) are determined by the recipes configured in the target Radius environment, allowing for flexible deployment to diverse compute platforms without altering the core application logic or Bicep code. | ||
|
|
||
| > Note that Radius will have to merge the configurations from the environment and container definitions with the configurations provided within the `platformOptions` property. For example, if the `platformOptions.kubernetes.metadata.labels` is specified in the container definition, it will override the labels defined at the environment level. |
There was a problem hiding this comment.
See comments above. I think there is a false assumption here that Radius has a notion of what a pod spec is. In fact, Radius has no knowledge of the pod spec schema—only the recipe does.
| * The developer deploys the application using `rad deploy app.bicep --environment my-env`. | ||
| * Radius uses the registered recipe for `Applications.Core/volumes@2025-05-01-preview` to provision the storage volume and the recipe for `Applications.Core/containers@2025-05-01-preview` to deploy the container with the storage mounted as a volume in the container. | ||
|
|
||
| #### User Story 11: As a platform engineer, I want a local development experience for Recipes and RRTs, so that I can iterate on custom Recipes without needing to publish them to an OCI registry: |
There was a problem hiding this comment.
I recommend pulling this out into it's own standalone feature spec.
Co-authored-by: Zach Casper <[email protected]> Signed-off-by: Will <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
…com/willtsai/design-notes-radius into compute-extensibility-feature-specs
Signed-off-by: Will Tsai <[email protected]>
Add feature spec for compute extensibility to define requirements and user experience